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ABSTRACT 


In response to technological advances. Network Centric 
Warfare (NCW) emerged as a theory to leverage the 
technology available in today's world. Advocates of NCW 
claim that technology will improve information sharing by 
"...robustly networking a force", thereby improving mission 
effectiveness. This study proposes a methodology with 
which to test the first tenet of NCW: a robustly networked 
force improves information sharing. 

Lessons learned from Human Systems Integration (HSI) 
demonstrate that in order to improve mission effectiveness, 
characteristics of both the human and the technology must 
be considered. As such, the impact of human 
characteristics and traits on mission effectiveness, as 
measured by individual and team performance, are assessed 
using a computer simulation, C3Fire . 

Results at the individual level suggest that persons 
scoring high on extraversion and low on pessimism perform 
better than those scoring low on extraversion and high on 
pessimism. In contrast, at the team level, homogenous 
teams as measured by optimism-pessimism perform worse than 
diverse teams. Results of this thesis provide a 
methodology with which to examine NCW's claims in a 
laboratory setting. Preliminary evidence demonstrates the 
need to consider human characteristics and traits in the 
design and composition of network teams. 
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EXECUTIVE SUMMARY 


Societal progression from the Industrial to the 
Information Age has produced the need for transformation in 
warfighting. In response to technological advances. 

Network Centric Warfare (NCW) emerged as a theory to 
leverage the technology available in today's world. 
Specifically, advocates of NCW claim that technology will 
improve information sharing by "...robustly networking a 
force", thereby improving mission effectiveness. However, 
just as with any new theory, a methodology and procedure 
must be in place for validating its claims. This study 
addresses this gap by proposing a methodology with which to 
test the first tenet of NCW: a robustly networked force 
improves information sharing. 

Lessons learned from Human Systems Integration (HSI) 
demonstrate that in order to improve mission effectiveness, 
characteristics of both the human and the technology must 
be considered. As such, the impact of human 
characteristics and traits (i.e., personality, cultural 
adaptability, and optimism-pessimism) on mission 
effectiveness, as measured by individual and team 
performance, are assessed using a computer simulation, 
CSFire. 

Results at the individual level suggest that 
participants scoring high on extraversion and low on 
pessimism perform better than those scoring low on 
extraversion and high on pessimism. In contrast, at the 
team level, homogenous teams as measured by optimism- 
pessimism, perform worse than diverse teams. Results of 
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this thesis provide a methodology in which to examine NCW's 
claims in a laboratory setting. Findings from this study 
demonstrate the need to consider human characteristics and 
traits in the design and composition of network teams. 
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1. INTRODUCTION 

A. PROBLEM STATEMENT 

Fundamental changes in technology, weapon systems, and 
marketplace commodities functioning as tools of war are 
affecting the very character of warfare. In response to 
these changes, the United States Department of Defense 
(DoD) is transforming. This transformation encompasses how 
technology and new operational concepts and structures can 
be used to augment military preparedness (Garstka, 2003). 

In addition, this recent transformation of the DoD has 
embraced Network Centric Warfare (NCW) as an emerging 
theory of warfare (Cebrowski, 2002) . 

In theory, NCW accelerates the ability to respond to 
uncertainty in dynamic situations by relying on human 
networks. Advocates of NCW claim that adopting NCW will 
improve information exchange and sensemaking which are 
referred to as macrocognitive functions. In theory, 
effectiveness in macrocognitive functions increases because 
complex cognitive systems would enable people to coordinate 
more efficiently across time, space, and organizational 
boundaries. However, claims of the benefits of NCW have 
not been systematically verified in field or laboratory 
settings. 

One approach to verifying improvements in information 
exchange and sensemaking is the study of macrocognition. 
Macrocognition describes mental activities that must be 
successfully accomplished to perform a task or achieve a 
goal (Klein et al., 2003) . Macrocognition is a term used 
to indicate the level of description of the cognitive 
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functions that are performed in natural versus artificial 
laboratory settings (Cacciabue & Hollnagel, 1995). As part 
of macrocognitive functions, problem detection, uncertainty 
management, and situated cognition occurs. These processes 
are generally performed in collaboration by a team working 
in a natural setting. 

In theory, NCW is one step towards military 
transformation. However, a methodology for assessing NCW's 
benefits in macrocognitive performance has not been 
established. In some cases, researchers have chosen to 
explain variability in performance through individual 
dispositions such as personality. Therefore, other factors 
such as individual characteristics and traits may also 
affect performance. 

B. OBJECTIVES 

The objective of this research effort is to design and 
establish a laboratory and research method focused on 
evaluating the claims of NCW from a human systems 
integration (HSI) perspective. The specific goals of this 
effort include: 

• To develop and implement a laboratory for 
evaluating team performance . 

• To design and test a set of procedures for 
applying individual and team performance 
measures. 

• To evaluate the tenets and claims of NCW from a 
HSI perspective. 

• To demonstrate the utility of the Dynamic Model 
of Situated Cognition (Miller & Shattuck, 2005) 
in providing a descriptive analysis in team 
effectiveness and its applicability in future 
research. 
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C. RESEARCH QUESTIONS 

The specific research questions addressed in this 
study include: 

• Are there individual characteristics and traits 
that relate to individual performance? 

• Are there individual characteristics and traits 
that relate to overall team performance? 

• Do diverse teams perform differently from 
homogenous teams? 

• Are there performance differences when blocking 
on various team member traits in team design? 


D. BACKGROUND 

As cited in Office of Force Transformation (2005) , 
former Secretary of Defense Paul Wolfowitz (2001) stated: 

Throughout history, warfare has assumed the 
characteristics of its age and the technology of 
its age. Today we see this trend continuing as we 
move from the Industrial Age warfare with its 
emphasis on mass to Information Age warfare, 
which highlights the power of networked 
distributed forces, p.7 

The characteristics of a networked distributed force 
are particularly evident in coalition command and control 
(C2) operations. Coalition C2 staffs are expected to use 
networks to share information among diverse members. C2 
staffs involve multinational team members characterized by 
divergent national interests. Differences in culture, 
organizational affiliation, and professional backgrounds 
are believed to affect macrocognitive functions and 
processes such as uncertainty management (Klein, Pongonis & 
Klein, 2000). 
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To capitalize on the perceived advantage of networked 
distributed teams, the U.S. military has adopted NCW as an 
emergent theory. However, with any new theory there is the 
challenge of demonstrating its utility and worth. If NCW 
is to be fully embraced, empirical and qualitative research 
must provide support for its claims. 

Proponents of NCW would argue that NCW is required for 
effective military transformation. In theory, mission 
effectiveness is enhanced when information sharing occurs 
as a result of "robustly networking a force." As part of 
networking a force, Alberts and Hayes (2003) emphasize 
interoperability, which is the ability to work together. 
However, our understanding of technology and its 
integration into military operations to enhance mission 
effectiveness is incomplete. A portion of this gap between 
leveraging technology to enhance mission effectiveness 
includes the dynamic interaction of team members and an 
understanding of how individual characteristics and traits 
contribute to team interaction. 

Decades of team research have been conducted, but the 
findings do not provide the evidence needed to accept or 
refute NCW claims. Research at the macrocognitive level is 
needed to test NCW assumptions. Salas et al. (2003) 

reviewed twenty-five years of research on team 
effectiveness. They identified seven areas needing 
research. Two of these are: (1) the need to understand 
distributed teamwork, and (2) the need to focus on team 
culture. 
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E. HUMAN SYSTEMS INTEGRATION 

In this thesis, we investigate the claims of NCW from 
an HSI perspective. HSI is a multidisciplinary field with 
eight basic areas or domains of study: 

• Manpower 

• Personnel 

• Training 

• Human Factors Engineering 

• Health Hazard 

• System Safety 

• Personnel Survivability 

• Habitability 

We focus on three of these eight domains: personnel, 
training, and human factors engineering (HFE) . Personnel 
and training include aspects of selection and 
classification; physical, cognitive, and educational 
characteristics; knowledge, skills, and abilities; and, 
finally, simulation and virtual environments. HFE is the 
integration of physical and mental limits, biases, and 
behaviors into system definition, design, development, and 
evaluation to optimize human-machine performance (Lockett & 
Powers, 2003) . The personnel, training, and HFE domains of 
HSI provide a foundation by which claims of NCW can be 
tested. 
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F. THESIS ORGANIZATION 

In the next chapter, NCW theory and literature 
pertaining to factors that influence macrocognitive 
functions are described. Since a methodology for testing 
NCW's claims has not been previously created. Chapter III 
provides detail regarding hardware/software use in the 
laboratory setting. This is followed by a description of 
the pilot study in Chapter IV. Finally, Chapters V and VI 
present the results, conclusions, and recommendations from 
the study. 
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n. LITERATURE REVIEW 


The literature review is divided into multiple 
sections. The first section addresses the theory and 
claims of NCW. Other sections focus on personality, 
cultural adaptability, and optimism-pessimism. Finally, a 
conceptual model is described to demonstrate how 
personality, cultural adaptability, and optimism-pessimism 
influence how teams perform. 

A. NETWORK-CENTRIC WARFARE 

NCW has become synonymous with recent attempts by the 
DoD and other military forces to leverage the networking 
capabilities of the information age. Cebrowski and Garstka 
(1998), in one of the earliest references to NCW, state 
that "...NCW derives its power from the strong networking 
of a geographically dispersed force." NCW is about human 
behavior within a network environment. The word "network" 
(i.e., as a noun) refers to the information technology and 
can only be an enabler. Used as a verb, "to network" 
implies human behavior, the action and the main focus of 
NCW (Office of Force Transformation, 2005). 

Descriptions of NCW concepts are found in Power to the 
Edge (Alberts & Hayes, 2003) . Stemming from the work of 
others, they list four tenets that describe the U.S. 
military's approach to NCW: 

• A robustly networked force improves information 
sharing. 

• Information sharing and collaboration enhances 
the quality of information and shared situational 
awareness. 
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• Shared situational awareness enables 
collaboration and self-synchronization and 
enhances sustainability and speed of command. 

• These in turn dramatically increase mission 
effectiveness. 

These tenets facilitate the U.S. military's 
implementation of NCW across four domains: physical, 
information, cognitive, and social. The physical domain 
encompasses operations such as strike, protect, and 
maneuver. The information domain includes activities of 
information sharing, creation, and manipulation. The 
cognitive domain embodies perceptions, awareness, beliefs, 
and values. Finally, the social domain recognizes 
interactions between and among organizational elements. 
Miller and Shattuck (2005) refer to a fifth domain, the 
ecological domain. The ecological domain has been recently 
proposed by Lindh (2004) to address the domain of context. 

As Figure 1 illustrates, the domains of warfare 
conceptualized in U.S. military doctrine overlap with one 
another. An analysis of each domain of warfare is beyond 
the scope of this research. However, the interaction of 
the social and cognitive domains (later referred to as 
socio-cognitive) comprise information exchange and 
sensemaking (Alberts & Hayes, 2003), the main focus of this 
research. 
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Figure 1. Domains of Warfare Overlapping (From: Office of 

Force Transformation, 2005, p. 21) 


The socio-cognitive domain comprises macrocognitive 
behaviors such as information exchange and sensemaking 
(Figure 2, NOW Conceptual Framework). The NOW Conceptual 
Framework is the result of a 2002 workshop on NOW and 
networked enabled capabilities (Evidenced Based Research, 
2003). This framework emphasizes behaviors which are 
believed to be critical in macrocognition. Behaviors such 
as sensemaking are addressed at the individual and team 
level (Alberts, Hayes, & Signori, 2001). 

At the team level, macrocognitive behaviors include 
the degree to which (1) information is shared and (2) 
shared awareness is achieved. Proponents of NOW believe 
that information and networking "...form the center of 
Network Centric Cperations" (Evidenced Based Research, 

2003, p. 32). The extent to which teams are networked (the 
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quality of networking) along with their ability to rapidly 
share information (the quality and degree of information 
sharing) are said to be critical in determining overall 
effectiveness. Sensemaking involves activities that allow 
individuals to "make sense" of information in the context 
of experience, use this information to make inferences, and 
organize information into decisions (Evidenced Based 
Research, 2003). Alberts and Hayes (2003) state "...these 
variables are at the heart of the collaborative processes 
that NCW seeks to exploit" (p. 99). 



Figure 11: Top-Level NCO Conceptual Framework, Version 1.0 

Figure 2. NCW Conceptual Framework (From: EBR, 2003, p. 4) 

B. PERSONALITY 

The study of personality in characterizing team 
effectiveness or understanding military adversaries dates 
back to World War II. During this time, personalities of 
enemy nations including Germany, the Soviet Union, and 
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Japan were studied by anthropologists at the direction of 
the U.S. government (Hofstede & McCrae, 2004). Trait 
psychology emerged as a contemporary approach to the study 
of personality. Trait psychology focuses on the assessment 
of individual differences in enduring dispositions as 
predictors of behavior. Dispositions such as personality 
are one method by which researchers have chosen to explain 
variability in team performance. 

Central to the study of trait psychology and the 
characterization of personality is the development of the 
Five-Factor Model (Digman, 1990). The Five-Factor Model 
emerged as a result of over 70 years of systematic research 
beginning with efforts to organize the language of 
personality (John et al ., 1988; Digman, 1990). Klages 
(1926) suggested that a careful analysis of language would 
assist in the understanding of personality which lead to 
Baumgarten's (1933) examination of personality terms 
commonly found in the German language (as cited in Digman, 
1990) . 

There have been other taxonomies of personality 
encompassing as few as three factors - Psychoticism, 
Neuroticism, and Extroversion/Introversion (Eysenck, 1960) 
and as many as 16 factors (i.e., Cattell's Sixteen 
Personality Eactor System; Cattell, 1965). As cited in 
Digman (1990), the work of Eisk (1949) and Tupes and 
Christal (1961) helped to establish the superiority of a 
five-factor approach. Replicated by later researchers (for 
a review see Digman, 1990), the Eive-Eactor Model has been 
recognized for its robustness. Any model for structuring 
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individual differences will have to encompass the Big Five 
dimensions at some level (Digman, 1990). 

The Five-Factor Model consists of five dimensions of 
personality (Digman, 1990), often referred to as the Big 
Five. Personality researchers have failed to reach a 
consensus on the names representing each dimension . For 
this thesis, Costa and McCrae's (1985) five personality 
constructs are used. The five constructs are known as 
extraversion, agreeableness, conscientiousness, 
neuroticism, and openness. Within each dimension, each 
construct is defined by at least six specific traits or 
facets which are addressed in further detail later in this 
section. 

Research within the last few years has provided 
support to the utility of using personality measures as 
predictors of performance. McHenry et al., 1990 (as cited 
in Neuman & Wright, 1999) observed that personality is a 
predictor of job performance beyond general cognitive 
ability and job-specific skills for Army personnel. The 
following sections are intended to provide a more detailed 
explanation of each of the Big Five and its implications 
for assessing individual and team performance. 

The Big Five factor of Conscientiousness is comprised 
of competency, dutifulness, need for achievement, self- 
discipline, and the tendency to think carefully before 
acting (Barrick & Mount, 1991). According to Barrick and 
Mount (1993), conscientiousness is related to individual 
performance regardless of task requirements. Later 
research (Kichuck & Wiesner, 1997; Neuman & Wright, 1999) 
suggested a relationship between conscientiousness and team 
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performance. Teams, whose members score high on the "need 
for achievement", one facet of conscientiousness, 
outperform teams whose members scored low (see French, 

1958; Schneider & Delaney, 1972; Zander & Forward, 1968). 

Agreeableness is characterized by trust, 
straightforwardness, altruism, compliance, modesty, and the 
ability to be tender-minded (Barrick & Mount, 1991) . In 
their review, Neuman and Wright (1999) suggest that 
agreeableness is a predictor of job performance at the 
individual level (see Rose et al., 1994; Tett, Jackson, & 

Rothstein, 1991) . At this level, facets of agreeableness 
such as trust, straightforwardness, altruism, and 
compliance are traits that are desirable for the social 
interactions found among team members (Aronoff & Wilson, 
1985) . Teams characterized by agreeable members are 
expected to exhibit interpersonal skills such as ability to 
resolve conflict and communicate openly in a manner which 
promotes information exchange and sensemaking (Aronoff & 
Wilson, 1985; Neuman & Wright, 1999; Agarwal, 2003). In 
their analysis of the human dimension of NCW, Warne et al. 
(2004) cite research which demonstrated that the extent to 
which an individual trusts another significantly impacts 
their willingness to share valuable information with others 
(see Fine & Holyfield, 1996). 

Extraversion is characterized by sociability, 
gregariousness, assertiveness, talkativeness, and 
activeness (Barrick & Mount, 1991) . Kickuk and Wiesner 
(1997) cite research showing that extroversion is 
positively related to team performance (Haythorn, 1953; 
Ghiselli & Lodahl, 1958; Smelser, 1961; Altman & Haythorn, 
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1967; Bouchard, 1969; Shaw & Harkey, 1976; Driskell et al., 
1987; Williams & Sternberg, 1988) and participation with 
the team (Mann, 1959; Watson, 1971). Extroverts are said 
to help facilitate intra-team communication and are 
generally ready to share information (Bradley & Herbert, 
1997) . 

Neuroticism is characterized by traits such as 
anxiety, depression, anger, embarrassment, emotionality, 
and insecurity (Barrick & Mount, 1991). Neuroticism has 
also been thought of as a lack of emotional stability or 
adjustment (Digman, 1990) . Neuroticism has not been found 
to be related to performance (Kickuk & Wiesner, 1997). 
However, after reviewing studies completed by Haythorn, 
(1953), Man (1959), Shaw (1971) and Thomas et al. (1996), 

they posit that Neuroticism may be negatively correlated 
with team performance. 

Traits associated with Openness include imagination, 
culture, curiosity, originality, broad-mindedness, 
intelligence, and having an artistic ability (Barrick & 
Mount, 1991). In a review by Kichuk and Wiesner (1997), 
openness to experience is said to be predictive of an 
individual's training proficiency but not predictive of 
his/her performance. The relationship between openness and 
overall performance is inconclusive (Kichuk & Wiesner, 

1997) . 

C. CULTURAL ADAPTABILITY 

Culture has often been used to explain human behavior, 
but "...there is no single, accepted definition" (Bird, 2001, 
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p.2). However, there are characteristics which help 
conceptualize culture. Bird (2001) identifies four common 
characteristics: 

1. Culture includes systems of values; 

2. Culture is learned, not innate; 

3. Culture distinguishes one group from another; and 

4. Culture influences beliefs, attitudes, perception 
and behavior in 'somewhat' uniform and 
predictable ways. (pp. 2-3) 

In the context of NCW, the study of culture may be 
best viewed as a lens (Trandis, 1994) . The ability to 
examine culture through a lens provides the flexibility to 
study culture at both an individual and team level . This 
approach also allows researchers to examine the dynamic 
relationships that may exist between the four previously 
mentioned characteristics of culture. 

Cultural differences affect coordination and 
communication, which are factors of sense-making. In their 
review of culture and team performance, Powell, Piccoli, 
and Ives (2004) suggest that differences in culture may 
lead to coordination difficulties (see Johansson et al., 
1999; Kayworth & Leidner, 2000; Maznevski & Chdoba, 2001; 
Robey et al., 2000) and create obstacles to effective 
communication (see Kayworth & Leidner, 2000; Barker & 

Sahay, 2002; van Ryssen & Godar, 2000). 

Cultural differences within teams may also be 
magnified in network-centric environments. Networking 
implies linking individuals together to work in teams. 

Thus, networking, from a human sense, ultimately results in 
teamwork. Teamwork connotes individual accountability, 
information exchange, and a general sense of working 
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together (Drucker, 1999). Consequently, teamwork is viewed 
as a skill and its absence can be a barrier to effective 
performance (Warne et al. , 2004) . 

Klein, Pongonis, and Klein (2000) identify five 
cultural differences that can disrupt sensemaking, 
decision-making, coordination, and communication in 
military team work: (1) power distance, (2) dialectical 
reasoning, (3) counterfactual thinking, (4) risk assessment 
and uncertainty management, and (5) activity orientation. 
This research focuses on only one of these five 
differences, uncertainty management. 

Uncertainty is "the state of being unsettled or in 
doubt" (American Heritage Dictionary, 2000). Uncertainty 
can be a function of ambiguous information, perceived 
threat, or harmful situation. Greco and Roger (2001) 
advocate that "...stress under these conditions varies as a 
function of an individual's efforts of, appraisal of, and 
coping with, the event" (p. 517). The study of uncertainty 
and human behavior has become known variously as tolerance 
of ambiguity, uncertainty avoidance, anxiety, and risk 
avoidance. These terms are used interchangeably in this 
thesis . 

Uncertainty has been conceptualized and studied within 
a variety of cultural contexts. At the national culture 
level, uncertainty may be termed uncertainty avoidance. 
Cultures classified as strong in uncertainty avoidance are 
active, aggressive, emotional, compulsive, security¬ 
seeking, and intolerant. Other cultures that can be 
characterized as low in uncertainty avoidance are 
contemplative, less aggressive, unemotional, relaxed. 
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accepting personal risks, and relatively tolerant 
(Hofstede, 1991). Uncertainty avoidance has also been 
"related to anxiety, need for security and dependence upon 
experts" (Hofstede, 1980, p.llO). 

The need for certainty is not only different between 
cultures, but also within cultures. As a trait, the need 
for certainty has been labeled as an intolerance of 
ambiguity. Intolerance of ambiguity has been defined as 
"the tendency to perceive (i.e., interpret) ambiguous 
situations as sources of threat" (Budner, 1962, p. 29). 
Ambiguous situations perceived as threats include new 
technology, dynamic and complex environments, asymmetry of 
the battlefield, and dynamic information exchange networks. 
Overall, a high need for certainty or a low tolerance for 
ambiguity, imply "a preference for familiarity, symmetry, 
definiteness, and regularity" (Bar-Tal, 1994, p. 45) . 

Stress is one of the human responses to uncertainty. 
Stress can be initiated by a distinct event, ambiguous and 
incomplete information, or a perceived threat. Stress also 
has an effect on performance. Performance differences 
which may result from stress include premature reactions, 
restricted use of relevant cues, more errors on cognitive 
tasks, and increased use of schematic or stereotyped 
judgments (see Bar-Tal, Raviv, & Spitzer, 1999) . However, 
not everyone's reactions to stressors are the same. 

A 12-month team performance study led by Sutton and 
Pierce (2003), found that uncertainty affected situation 
assessment (i.e., information exchange), coordination 
(i.e., response sequencing), and general support behavior 
(i.e., activity monitoring). Their study assessed the 
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degree to which cultural cognitive dimensions impact 
teamwork in a multinational headquarters. Specifically, 
teams comprised of individuals scoring high on need for 
certainty (uncertainty avoidance) engaged in behaviors that 
were different from team members with a low need for 
certainty. Individuals scoring high on need for certainty 
produced much more detailed information in the situation 
assessment phase of planning and coordination. These 
individuals also developed well-defined processes for 
information exchange and coordination. 

D. OPTIMISM AND PESSIMISM 

Pessimism is defined as a general tendency to have 
negative expectations. "Optimism is a disposition 
inclining one to positive expectations; pessimism inclines 
one to negative expectations" (Helton et al., 1999, p. 

311). Optimists and pessimists respond to stress in 
different ways. Stress narrows attention and causes 
important information to be overlooked. This narrowing 
affects the problem solving process, another process 
related to macrocognition (Hoffman, Roesler, & Moon, 2004). 
Central to the activity of problem solving is sensemaking 
(Russell et al., 1993). Seligman (as cited in Grasha, 

2000) suggests that optimists are willing to challenge 
problems, persist until a solution is found, and involve 
others in resolving issues (Seligman, 1991). Optimists are 
believed to participate more in information exchange. 

E. MODEL OF DYNAMIC SITUATED COGNITION 

NCW advocates cite the need for a process model to 
assess NCW's claims. However, such a process model has not 
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been available until recently. Miller and Shattuck (2005) 
proposed filling this gap using their Dynamic Model of 
Situated Cognition (DMSC) (Figure 3). 


The Dynamic Model of Situated Cognition 



Figure 3. The Dynamic Model of Situated Cognition (From: 

Miller & Shattuck, 2004) 

The DMSC emerged as an attempt to illustrate the 
relationships between technological systems and human 
perception and cognition. Since its creation, the DMSC has 
been used to describe what happens in operational 
environments with people who are engaged in goal directed 
behavior (see Miller & Shattuck, 2004, 2005). It has also 
been adapted by Miller and Shattuck (2005) as a process 
model for NCW (Figure 4). The DMSC can be used to examine 
macrocognitive processes and functions of information 
exchange and sensemaking which reside in the socio- 
cognitive domain. Situated cognition describes the fact 
that macrocognitive functions are generally performed in 
collaboration by a team working in a natural setting. 
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Accordingly, the left side of the model receives the 
greatest attention in this research. 

The Dynamic Model of Situated Cognition 
with NCOW Domains of Warfare 


Physical and Information Domains Cognitive Domain 



Figure 4. The DMSC adapted to the NCOW Domains of Warfare 

(From: Miller & Shattuck, 2005) 




The DMSC consists of a series of lens (A, B, and C, 
see Figure 3). Lens A, the lens between Ovals 3 and 4, 
directs attention to selected incoming stimuli. These 
stimuli are, in most cases either visual or auditory. 
Between Ovals 4 and 5 is Lens B. Lens B is believed to 
influence how data are organized into information. The 
lens between Ovals 5 and 6, Lens C is believed to guide the 
process of extrapolating current information (Shattuck & 
Miller, 2005) . 

Miller and Shattuck (2005) state that there are at 
least six classes of information embedded in the lenses 
that influence macrocognitive processes. The six classes 
of information are: 1) individual states and traits, 2) 
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social factors, 3) local context, 4) plan, 5) guidelines, 
and 6) experience. Two are applicable in the context of 
this research: individual states and traits, and social 
factors. Individual states and traits represent relatively 
enduring (e.g., personality) and transient (e.g., fatigue) 
characteristics of an individual. For a more detailed 
discussion of human trait and state measurements, see 
Miller et al. (2003). Social factors include issues 

ranging from team dynamics (e.g., homogeneity) to cultural 
differences. Together these two factors may significantly 
influence what is perceived by team members (Oval 4). 

The ovals to the right of lens A (Oval 4, 5, and 6) 
represent perception, comprehension, and predictions - 
loosely referred to as macrocognitive processes. 

Perception (a process occurring in Oval 4) is achieved 
through active and passive processes such as information 
exchange. Active processes refer to data requested or 
"pulled" by team members. Passive processes refer to data 
provided or "pushed" to team members. Comprehension is 
represented in Oval 5. Comprehension is a cognitive process 
described by terms such as fusion, integration, analysis, 
explanation, interpretation, and pattern recognition 
(Endsley, 1995). Comprehension is loosely referred to as 
sensemaking. Dervin's theory of sensemaking, (as cited in 
Salvolainen, 1993) is defined as: "... behavior, both 
internal (i.e., cognitive) and external (i.e. procedural), 
which allows the individual to construct and design their 
movement through time-space" (p.l6). Finally, Oval 6 
represents projections of individual team members. These 
projections are based on what have been comprehended (Oval 
5), and the affect of an individual's lens (Lens C). These 
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same lens components (i.e., individual states and traits 
and social factors) are believed to influence 
macrocognitive processes such as information exchange and 
sense-making. 

NCW, by its very nature, involves multiple actors, 
both human and machine. Miller and Shattuck (2005) state 
"As we move from an individual to the network of 
individuals that characterize NCW" the technological (or 
physical and information domains) remain the same. Figure 
5 below illustrates how ground truth (Oval 1) and data 
detected by sensor systems (Oval 2) will remain the same 
for each individual. However, Oval 3 through Oval 6 will 
differ for each of the three individual team members. As 
an illustration, the lines from individuals A and C to 
individual B represent the way individuals A and C convey 
information to individual B. The same descriptions of 
lenses and ovals are applicable in this illustration. 



Figure 5. DMSC Applied to Teams (From: Miller & Shattuck, 
2005, Published in Conference Proceedings 
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Situated cognition is a dynamic, ongoing process 
(Clancey, 1997). Figure 6 further represents this ongoing 
process by its emphasis on feedback in situated cognition. 
The feedback loops shown in Figure 6 provide insight into 
macrocognitive processes of networked individuals. The 
feedback loops flow from Oval 5 (comprehension) to Ovals 1, 
2, 3, 4 (environmental, sensors, 02 workstation, and 
perception). Feedback loops from Oval 6 (projection) also 
flow to Ovals 1, 2, 3, 4, and 5. Additional feedback loops 
exists in the model as well. These feedback loops extend 
from Oval 5 (comprehension) to the lenses and from Oval 6 
(projection) to the lenses. Miller and Shattuck (2005) 
have postulated that the lenses are dynamic and constantly 
change. 


A Dynamic Model of Situated Cognition 



Figure 6. DMSC Feedback Loops, (From: Miller & Shattuck, 

2004, p.5) 
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The factors influencing effective macrocognitive processes 
must be investigated insofar as they underpin mission 
effectiveness (Warne et al. , 2005). 
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III. DESIGN AND IMPLEMENTATION OE THE TEAM 
PEREORMANCE LABORATORY 


A. BACKGROUND 

Much of the time involved in this thesis effort was 
spent designing, constructing, and configuring the Team 
Performance Laboratory. The Team Performance Laboratory 
uses computer software to assess and evaluate human factors 
considerations in team performance. The Human Systems 
Integration Laboratory (HSIL) was reconfigured to 
accommodate two semi-private four person laboratories. 
Additionally, four mobile desktop stations were configured 
for easy set-up and take-down given limited laboratory 
space. A two person observer station was set up to 
network, control, observe, and collect data on the eight 
Pentium 4 desktop computers used by team members in the 
CSFire simulation. 

B. SOFTWARE 

Testing the claims of NCW using a computer generated 
synthetic environment required reviewing numerous game- 
based software packages. These included Mission to Mars, 
AWACS-AEDGE, Never Winter Nights - SABRE, and CSEire. 

Mission to Mars is a computer-generated interactive 
simulation currently being used by researchers to evaluate 
distributed interactive communication between dispersed 
elements. The simulations are developed around resource 
and time management themes within the context of geologic 
exploration expeditions on the surface of Mars. Groups of 
three-person crews are expected to operate a simulated 
exploration vehicle, an "Orbiter", "Lander", or "Rover", 
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via an individual workstation. The task is designed to 
last up to four hours. Mission to Mars automatically 
records both crew performance and psychosocial 
communication interactions (Hienz et al. , in press). 

The Airborne Warning and Control System - Agent 
Enabled Decision Guide Environment (AWACS-AEDGE), developed 
using 21®'^ Century Systems Inc.'s AEDGE™ infrastructure is a 
distributed, real-time team decision support environment. 

It is comprised of simulators, intelligent agents and user 
interfaces (Barnes, Petrov, & Elliott, 2002) . The AWACS- 
AEDGE was developed to represent core characteristics of 
the Airborne Warning and Control Systems (AWACS) Weapons 
Director (WD) team. Core characteristics of the AWACS-WD 
team include providing airborne surveillance, control, and 
communications functions for tactical and air defense 
forces. The AWACS-AEDGE was developed as an agent-based C2 
team decision support platform for research and training 
(Petrov et al., 2002). 

SABRE (Situation Authorable Behavior Research 
Environment) is a joint Defense Modeling and Simulation 
Office and Air Force Research Laboratory project. 

Developed by BBN Technologies, SABRE is a tool intended for 
team behavior research. SABRE is marketed as an aid for 
investigating general aspects of teamwork such as group 
decision-making, resource management, and information 
sharing. In addition, context-specific behaviors such as 
negotiating and accommodating mission-irrelevant requests 
for assistance are explored. This test-bed has been 
developed primarily to study the effects of personality and 
culture on behavior and performance in a cooperative team 
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mission (Leung, Diller, & Ferguson, 2005). Training time 
prior to data collection is approximately 2.5 hours (Warren 
et al., 2005) . 

The CSFire microworld is a command, control, and 
communications firefighting simulation. CSFire has been 
used for training and experimentation of team decision¬ 
making and team situational awareness (Granlund et al., 
2001; Granlund, 2002). CSFire generates a task environment 
in which a team of four people cooperate to extinguish a 
fire (see Figure 7). The user interface consists of 
several basic elements: a geographic information system 
(GIS), a diary, and an e-mail system. In the center of the 
user interface is a map consisting of a 40 x 40 matrix of 
cells; a map legend, clock, e-mail tool, and a truck status 
panel. Using these features, players of CSFire control 
three types of trucks (firefighting, fuel, and water) and 
are responsible for working together via e-mail to 
extinguish the fire. Players need to maintain a picture of 
fuel and water states during the game. The game records a 
variety of performance data in the form of logs. Training 
time is approximately 2 minutes. 
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Figure 7. An example of CSFire Microworld Setting, (From 
CSFire Instructions Granlund, 2005, p. 1) 

After reviewing the concepts of NCW and given the 
short amount of training time, CSFire was chosen. CSFire 
was also selected because it had been previously used to 
test concepts of NCW (Johansson et al., 200S) . 


C. INSTALLATION 

CSFire was installed on ten desktop Pentium 4 
machines. In addition to the software, basic requirements 
for the team performance lab included: 

• Two adjacent areas (Team Performance Laboratory) 
measuring at least 14' X 14'. Each area should 
be fully isolated from other areas in the HSIL to 
protect against background noise. Each area may 
require as many as twelve surge-protected 
electrical outlets. 

• Three 10/100 Ethernet ports capable of networking 
four computers at once. 

• Ten Pentium 4 or S CPU with near S.20CHz, 1.00 CB 
of RAM. 

• Ten standard "qwerty" keyboards and two button 
mouse. Ten monitors of the same size are needed. 
Monitors should be no smaller than a 17-inch 
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Diagonal and 16-inch V.I.S., CRT, 1024x768 
Pixels. Ideally, 19" or 20" Flat Panel LCD 
Monitors are required for resolution and 
interface visibility. 

All machines need to be configured by the 
Information Technology Assistance Center (ITACS) 
to receive a constant IP Address and connect to 
the NPS Intranet. 
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IV. PILOT STUDY 


Identifying and measuring human performance both on an 
individual and team level is challenging. Team performance 
variables must be operationally defined, using the 
constructs identified for specific operations or procedures 
and the resultant measurements selected for this study. 

Key variables selected for this study and their operational 
definitions are provided below. A major benefit of CSFire 
and a deciding factor for its use in this study is its 
capability to automatically collect several kinds of 
performance data. The primary variable of interest in this 
thesis, information exchange (or information currency), is 
an indicator of overall NCW performance which has been 
previously identified in the literature. In the context of 
NCW, information exchange (information currency) is 
consistent with objective measures of information sharing 
and sensemaking at the team and individual levels. For 
more information on NCW variables in addition to the ones 
which have been identified below, see Appendix B. For 
CSFire capabilities, refer to www.C3Fire.org . 

A. VARAIABLES 

1. Dependent Variables 

a. Information exchange (information currency): 
communication lag time (in seconds) between 
sending and the receiver opening a message. 

b. Overall CSFire Performance - the number of 
cells in the 40 x 40 matrix of cells lost 
due to ineffective firefighting in a 17 
minute session. 
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2. Independent Variables 

a. Personality scores on each of five 
dimensions - measured using the NEO-FFI. 

b. Uncertainty Management scores on each of 
three scales: (1) Uncertainty Response 
Scale, (2) Need for Cognitive Structure, and 
(3) Ability to Achieve Cognitive Structure 

c. Optimism-Pessimism - total score from the 
OPI. 

d. Conflict Avoidance - total score from the 
Conflict Avoidance questionnaire. 

B. PARTICIPANTS 

Much of the previous research on teams has used 
undergraduate students from civilian institutions, thereby 
threatening generalizability of the studies for U.S. 
military forces. Therefore, the current study specifically 
sought participants from a population of military officers 
expected to perform in network-centric operations. 

Thirty-two NPS students and faculty members (average 
age = 34.1, standard deviation = 6.12) were assigned to 
eight teams who participated in the first phase of the 
exercise. Sixteen additional participants (both NPS 
students and faculty) participated but data from these 
latter participants are not included in this pilot study 
due to incomplete survey responses. 

The pilot study focused on the performance of eight 
teams. Of the eight teams, five were all male. Two teams 
were of mixed gender and one team was all female. 

Individuals in each team were known to each other. 
Naval Postgraduate School students enter as cohorts taking 
the same classes for 18 to 28 months, making it virtually 
impossible to avoid teams consisting of individuals who had 
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no prior experiences of working together. Consequently, 
each team was self-selected and entered the laboratory as a 
unit. 


The 32 participants represented most branches of 
service including civilians from the DoD. See Table 1 for 
more detailed demographic data. 


Navv 

Armv 

Marines 

Air Force 

Coast Guard 

DoD Civilian 

Total 

16 

6 

1 

3 

0 

6 

32 


Table 1. Military Service Representation 


C. INSTRUMENTS 

A variety of standardized surveys were used for 
collecting individual characteristics and trait data. The 
surveys selected for this research were a mix of open- and 
closed-ended questions that were 3-5 pages in length. For 
consistency and ease of scoring, many of the surveys were 
scored using a five-point Likert scale (1 = Strongly Agree 
to 5 = Strongly Disagree). 

The surveys listed below were selected because of 
their perceived ability to be related to team performance. 
The number of surveys selected was indicative of the 
absence of well defined individual characteristics and 
traits that are related to team performance literature (See 
Chapter II: Literature Review). Refer to the references 
listed for additional information on and a copy of each 
survey. 

1. Demographic Questionnaire 

The demographics questionnaire is a 15-item 
questionnaire with both open-ended and forced-choiced 
questions which assesses the participants' (1) personal. 
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academic, and work-related background, (2) international 
experiences, (3) experience in the military and teamwork, 
and (4) experience with computers, especially chat 
programs. The 15-item questionnaire was adapted from 
ongoing research on Bridging Cultural Barriers to 
Collaborative Decision Making in Onsite Operations 
Coordination Centers (K. Smith, personal communication, 
August 22, 2005, 11:32PM). 

2. NEO Five-Factor Inventory (NEO-FFI) 

Team member personality was measured using the NEO- 
FFI, a 60-item questionnaire. It is the shortened version 
of the Revised NEO-Personality Inventory (NEO-PI-R; Costa 
and McCrae, 1992). The NEO-FFI provides an accurate and 
concise measure of the "Big Five" domains of adult 
personality: extroversion, agreeableness, 

conscientiousness, emotional stability, and intellect (or 
openness) and their facets (Briggs, 1989). 

The NEO-FFI has adequate internal consistency, 
construct, and discriminative validity across diverse 
samples (Ball et. al., 2001; Costa & McCrae, 1992). The 
NEO-FFI Form S correlates with the NEO-PI-R domain scales 
at .77-.92 and has a internal consistency values ranging 
from .68-.86 (PAR, Inc., Retrieved 2005). Additionally, 
the NEO-FFI was chosen over other popular indices of 
personality because of its relational value to national 
cultural constructs previously mentioned in the literature 
(Hofstede & McCrae, 2004) and its 10-15 minute 
administration time. 
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3. Uncertainty Response Scale 

The Uncertainty Response Scale (URS; Greco & Roger, 
2001) is a 48-item scale which predicts individual 
differences in coping with uncertainty. The URS is 
comprised of three factors. Emotional Uncertainty (EU), 
Desire for Change (DEC), and Cognitive Uncertainty (CU). 

EU is the desire to which an individual responds to 
uncertainty with anxiety and sadness. DEC is the degree to 
which an individual enjoys novelty, uncertainty, and 
change. CU is the degree to which an individual prefers 
order, planning, and structure in an uncertain environment. 
Participants rate the degree to which a statement is true 
for themselves using a 5-point scale with endpoints being 1 
= Never and 5 = Always. Scores for subscales are 
determined by totaling the point value of statements 
associated with each subscale. Higher scores indicate a 
greater tendency toward maladaptive responses to 
uncertainty (EU), greater enjoyment of the unknown (DEC) , 
and greater preference for control under uncertain 
conditions (CU). The URS has an internal consistency of 
.89, .90, and .85 for subscales EU, DEC, and CU 

respectively, and test-retest reliability estimates of .79, 

.86, and .80. 

4. The Need for Cognitive Structure Scale 

The Need for Cognitive Structure Scale (NCS; Bar-Tal, 
1994) is a 20-item scale that assesses the extent to which 
an individual prefers using cognitive structuring to 
achieve certainty. Cognitive structuring (or 
categorization) helps create certainty by filtering out 
inconsistent or irrelevant information. Participants rate 
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the degree to which they disagree or agree with statements 
using a 5-point Likert scale with endpoints of 1 = Strongly 
Disagree and 5 = Strongly Agree. Responses are totaled to 
create an "overall need for cognitive structure" score. 
Higher scores indicate a greater need for cognitive 
structure. The NCS has an internal consistency of .82 and 
test-retest reliability of .85 (Bar-Tal, 1993, 1994). 

5. The Ability to Achieve Cognitive Structure Scale 

There are individual differences in the ability to 
effectively organize information to fit existing knowledge 
structures or to process information that is inconsistent 
with existing structures. The Ability to Achieve Cognitive 
Structure Scale (AACS; Bar-Tal, 1994) is a 24-item scale 
that assesses this trait. 

Participants rate the degree to which they disagree or 
agree with statements using a 5-point Likert scale with 
endpoints of 1 = Strongly Disagree and 5 = Strongly Agree. 
Responses are totaled to create an overall ability to 
achieve cognitive structure score. Higher scores indicate 
a greater ability to apply information processes that are 
consistent with an individual's level of NCS. The AACS has 
an internal consistency of .67 and a test-retest 
reliability of .86 (Bar-Tal, 1993, 1994). 

6. Optimism-Pessimism Instrument 

The Optimism-Pessimism Instrument (OPI) is a 56-item 
questionnaire. Optimism is a disposition representing a 
bias toward positive aspects of life. Pessimism is a 
disposition representing a bias toward negative aspects of 
life. In the OPI, 18 items indicate optimism (0), 18 items 
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indicate pessimism (P), and 20 items are filler. The scale 
was developed to measure the degree to which an individual 
is either an optimist or pessimist in their expectations 
(Dember et al., 1989). 

Respondents are asked to rate their agreement with the 
items using a four-point Likert-type scale with endpoints 
of 1 = Strongly Agree and 4 = Strongly Disagree. Unlike 
other measures of optimism-pessimism such as the 
Attributional Style Questionnaire (ASQ; Peterson et al. , 
1982) and the Life Orientation Test (LOT; Scheier & Carver, 
1985), OPI measures pessimistic and optimistic tendencies 
separately. 

The OPI has an internal consistency of .84 for 
optimism and .87 for pessimism. Test-retest reliability 
are .75 and .84 for optimism and pessimism respectively. 

7. Conflict Avoidance (CA) 

The Conflict Avoidance (CA) scale is a 23-item self- 
report measure that assesses a person's reaction to 
conflict. High scores on this survey indicate a tendency 
for wanting to avoid conflicts. The scale is comprised of 
items from the NEO-PI-R, the Intercultural Adaptation 
Potential Scale (ICAPS; LeRoux, J. & Matsumoto, D., 2000), 
the ROAD, and an additional Conflict Avoidance scale taken 
from (Tjosvold, 1985; Barker, Tjosvold, & Andrews, 1988). 

D. PROCEDURE 

Data collection was completed in two phases. In the 
first phase of data collection, eight teams of four NPS 
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faculty and students were asked to participate in a single 
90 minute exercise. Each laboratory exercise took place at 
the HSIL. 

At the beginning of each exercise, participants were 
asked to register and pick-up a manila folder containing 
their informed consent form and surveys. After each 
participant completed their informed consent form, they 
began completing three of the seven surveys. After 
completing the surveys, participants were asked to read the 
CSFire instructions. Once everyone completed their surveys 
and had a chance to read over the CSFire instructions, each 
of the four team members moved into position on a four 
person station. Participants played CSFire as a team 
training exercise for three minutes. Afterwards, 
instructions were reviewed and participants were given the 
opportunity to familiarize themselves with CSFire. During 
this S minute exercise, questions were encouraged and 
answered aloud to ensure participants were familiar with 
the game. 

When the training period ended, participants were 
given another opportunity to ask questions. Then, the 
first 17 minute data collection session began. At the end 
of the first seventeen minute session, participants were 
instructed to finish the remaining four surveys. After 
each participant answered all surveys, the final 17 minute 
CSFire session started. 

The first phase of this pilot study consisted of self- 
organized teams. Results from these teams were used to 
characterize teams based on individual characteristics and 
traits and to relate performance to team characteristics. 
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The second phase of this study used pre-selected teams 
designed by the researcher. These participants were asked 
to return for a second 17-minute play of CSFire. The 
original self-organized teams were replaced by teams 
designed on the basis of their individual characteristics. 
Four teams were constructed on the basis of agreeableness 
(High vs. Low) and cognitive uncertainty (High vs. Low). 

The personality trait of agreeableness was measured using 
the NEO-FFI. Individual responses to CU were collected 
using the Uncertainty Response Scale (URS). The NEO-FFI 
classifies individuals into one of five categories (Very 
Low, Low, Average, High, and Very High) for each 
personality trait, providing a mechanism for assessing the 
degree to which a personality trait is present. The URS 
scores an individual's degree to which they respond to 
uncertainty. Using all 32 responses on the URS, summary 
statistics were calculated. Upper and lower bounds were 
derived for responses to CU. 
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V. RESULTS OF PILOT STUDY 


A. SUMMARY STATISTICS OF PERSONALITY TRAITS 

The first step of this analysis summarizes individual 
characteristics and traits from the sample of NPS students 
and faculty. The sample consisted of thirty-two 
participants. Responses to the NEO-FFI, the personality 
measures used in the study, were organized into five- 
factors with five levels for each factor. Participants 
were classified as either "Very High" to "Very Low" in each 
of five personality factors of extroversion, openness, 
agreeableness, neuroticism, and conscientiousness. 
Distributions of each personality characteristic are given 
in Table 2 . Across the dimensions of personality, the 
classification "Average" was the most predominant with 
thirty-one percent of all responses. 



Extroversion 

Openness 

Openness 


Conscientiousness 


2 

2 

4 

6 

4 

Low 

2 

4 

7 

13 

8 


14 

11 

11 

8 

6 


10 

11 

8 

4 

11 


4 

4 

2 

1 

3 

Total 32 32 32 32 32 


Table 2. Classification of Personality Type and Strength 

of Classification 


Plots similar to Figure 8, are computed for each 
individual characteristic and trait measure (See Appendix 
E-F). These plots check for abnormalities in responses to 
each survey. As Figure 8 demonstrates, responses to each 
survey were generally varied and none of the responses 
stand out as anomalous. 
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Response to Uncertainty 


— Emotion Uncertainty 
Desire for Change 
Cognitive Uncertainty 


1 3 5 7 9 11 13 15 17 19 21 23 25 2 7 29 31 

Participants 

Figure 8. Scores on the Uncertainty Response Scale (URS) by 

Participant 

B. SUMMARY STATISTICS FOR PERFORMANCE 

Next we examine team performance. Figure 9 plots the 
lag time averaged over the number of messages sent in each 
exercise by team for both trial 1 and trial 2. It 
illustrates overall performances of seven of the eight 
teams. One team's performance is omitted because its 
members had more experience with actual play of the game. 
Time lag is a performance measure of information exchange 
and represents currency of information. Smaller time lag 
represents more current and relevant information. This 
variable is believed to be critical to team performance in 
dynamic environments. Having more current information may 
improve mission effectiveness in network environments. 

Thus, currency of information is a performance measure for 
mission effectiveness in NCW (Effective Based Research, 

2003) and the performance variable of choice in this study. 
In both trials. Team 4 had the smallest average time lag 
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and thus outperformed the other teams with respect to 
information exchange (Figure 9). 


Average Lag Time by Team 



Trial 1 
Trial 2 


Teams 


Figure 9. Information Exchange Between by Teams 

Overall Performance: 

Number of Cells Lost Due to a Lack of FireFighting 
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Figure 10. Overall Team Performance, Lost Cells 

As Figure 10 illustrates. Team 3 outperformed all 
other teams in overall performance. Overall performance is 
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defined as the number of cells lost due to ineffective 
firefighting. 

The analyses and graphs provided in the remaining 
chapters of this thesis are from trial 1 data only. No 
comparisons were made between trials due to learning 
effects and strategy changes within teams. As illustrated 
in Figures 9 and 10, Team S's average time lag increased 
between trials, but overall performance remained relatively 
superior among teams for both trials 1 and 2. Further 
analysis from communication logs showed that one member of 
Team 3 failed to check messages from other teammates until 
the final minutes of play in trial 2. Thus, Team S's 
average time lag increased and forced the remaining three 
teammates to reorganize quickly to cover for the lack of 
communication. With Team S's prior experience in trial 1, 
they were able to remain effective. Because the experiment 
was not designed to account for the effects of learning, 
summary data from trial 2 beyond what has previously been 
presented is not included. 

Individual performance drives team performance. 
Individual performance within teams may provide insights 
into variance in team performance. In each exercise, 
average time lag was recorded for each participant within 
each team. In CSFire, a message is hidden and therefore 
cannot be read until a player clicks the "next button". 

The assumption here is that when a player "clicks the "next 
button", the message is read. CSFire time stamps these 
events and stores who sends and reads their messages. The 
time between sending and receiving messages is termed time 
lag. In a seventeen-minute session, a player can send and 
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receive a number of messages. It is the responsibility of 
the player to ensure they remain current on information as 
it could affect individual and team performance, requiring 
that players check their mail regularly. For purposes of 
this study, information is not exchanged until the process 
of sending and reading a message is completed. 

Figure 11 shows individual performance differences in 
time lag. The box plot shows that team members in Team 4 
are less variable in managing information exchanges while 
members in Teams 2 and 5 are more variable. Following the 
assumption that individual performance is related to team 
performance. Team 2 was not expected to have the best 
performance; it did not. 



Green, Team 1 
Light Blue, Team 2 
Yellow, Team 3 
Purple, Team 4 
Orange, Team 5 
Red, Team 6 
Blue, Team 7 


Figure 11. Box plots of Message Time Lags by Participant 
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Time lag was also measured at the team level. Time 
lag measured at the team level is postulated to be 
indicative of team performance. For this study, 
sensemaking is assumed to be dependent on the currency of 
information provided and is therefore measured using time 
lag. Box plots in Figure 12 show differences in time lag 
at the team level. Team 4 is less variable in managing 
information whereas Team 2 shows more variability. 
Following the assumption that team time lag can affect 
overall team performance, overall performance was expected 
to be worse for Team 2 than or any other team. 



Figure 12. Box plots of Message Time Lag by Team 
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C. RELATIONSHIPS BETWEEN TEAM PERFORMANCE ASID PERSONALITY 

TRAITS 

The degree to which teams consist of members who are 
similar or dissimilar in individual characteristics and 
traits may also explain individual and team performance. 

The scatter plot in Figure 13 compares the degree to which 
a person is a pessimist against their individual 
performance (time lag). There appears to be a positive 
relationship between pessimism scores and time lag. The 
sample correlation r = .381 with a p-value of .05 for the 
two-sided test of no correlation. That is to say, less 
pessimistic individuals (those with lower scores) tend to 
manage information better than more pessimistic 
individuals. Additionally, there appears to be a negative 
relationship (r = -.391, p-value =.043) between individual 
performance and scores on extroversion. Figure 14. 


Scatter 


(A 

■o 

C 

o 

o 

a> 

(A 

200.000 

O) 

(0 

0) 

£ 

O) 

> 100.000 

< 

*5 

3 

*o 

> 

‘•u 

c 

0.000 


Figure 13. Pessimism vs Average Time Lag for Each Individual 
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Figure 14. Categories of Extroversion vs Individual 

Performance 

To explore the hypothesis that team member similarity 
is related to team performance as measured by time lag, 
standard deviations of characteristics and traits, as 
measured by the six instruments previously identified, are 
calculated by teams. Standard deviations provide another 
technique for characterizing teams based on individual 
characteristic and traits of its team members. For 
instance, in the category of neuroticism, a low standard 
deviation score for a team suggests that the team is 
similar in regard to neuroticism. Using this technique, it 
appears that team similarity in optimism is strongly 
negatively related to average team lag (r = -.786, p -value 
= .036) and that team similarity in pessimism is strongly 
related to the number of lost cells (r = .821, p-value = 
.036). These are the only measures of diversity related to 
the measures of team performance. See Figures 15 and 16 
below for the relationships between team similarity in 
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Standard Deviation 

Figure 16. Standard Deviation of Scores on Pessimism vs 

Number of Lost Cells 

A final step in this phase of the data analysis is to 
determine which individual characteristic and trait 
measures relate to performance and to each other. A 
correlation matrix was calculated to determine these 
relationships (See Appendix G). The individual 
characteristics and traits that might be related to 
performance are listed below. 


A. Individual Performance: 

• Pessimism is positively correlated with 

individual performance (time lag, r = .381, 
p-value = .05) 
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• Strengths of Extroversion is negatively 
correlated with individual performance (time 
lag, r = -.391, p-value = .043) 

B. Team Performance: 

• Optimism is negatively correlated with team 
performance (Average Team Lag Time, r= - 
.786, p-value = .036) 

• Pessimism is positively correlated with 
overall team performance (Number of Lost 
Cells, r = .821, p-value = .023) 

We note that because multiple comparison procedures were 
not used, these results will need to be tested in future 
research. 

The correlation matrix in Appendix G also provides 
some insight as to how measures of individual 
characteristics and traits are related to each other. 

There is evidence to conclude that each survey measures a 
separate and distinct individual characteristic and trait. 

D. RESULTS FROM PHASE TWO 

The results from the second phase of data analyses 
tests whether performance differs when team member traits 
are selected by design. Sixteen participants from the 
original 32 were asked to participate in this phase of the 
study. Four teams were constructed so that all members on 
each team had similar agreeableness scores or similar 
responses to uncertainty scores: 

• members of the Team "HA" had high agreeableness; 

• members of the Team "LA" had low agreeableness; 

• members of the Team "HURS" had a high need to 

resolve cognitive uncertainty; and 
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• members of the Team "LURS" had a low need to 
resolving cognitive uncertainty. 

Results for overall performance, as measured by total 
number of cells lost due to ineffective firefighting, are: 

• Team HA = 225 cells vs Team LA = 295 cells 

• Team LURS = 3 cells vs Team HURS = 314 cells 
With only one team of each type, we can not tell whether 
the differences in performance reflect actual differences. 
Thus, Phase Two of this pilot study is for illustrative 
purposes only. 

E. DISCUSSION 

There are performance differences for both teams and 
individuals. For each trial, the average lag time between 
messages (information exchange) was greater for Team 2 than 
for any other team. This difference indicates that team 
decisions were based on old information. Conversely, Team 
3 practiced information exchange behaviors that resulted in 
team decisions based on more current information. Thus, 
the average lag time between messages was less. On overall 
performance. Team 3 outperformed all other teams while Team 
2 lost the most cells due to fire. See Figures 19 and 20, 
screen shots from Teams 3 and 2 final performances, 
respectively . 
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Team 3, Trial 1 Play 
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Figure 17. CSFire Game Screen Shot Team 3, Trial 1 



Figure 18. 


CSFire Game Screen Shot Team 2, 


Trial 1 
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First, to determine if individual and team performance 
differences were related to individual characteristics and 
traits, we assumed there would be differences in individual 
characteristics and traits in the sample. The responses to 
each survey were generally varied and their distribution 
unimodal (See Appendix D-F). If survey data were uniform, 
or characterized by participants desiring to load their 
responses by answering "Strongly Agree" to every question, 
the distribution of responses would have shown distinct 
patterns. Therefore, the assumption that the sample 
consist of individuals with different characteristics and 
traits is upheld. 

Likewise, we expected that individuals and teams would 
exchange information at different rates (our measure of 
performance). As previously discussed, summary plots 
verified that these differences existed. To better 
understand these differences, box-plots like the one shown 
in Figure 12 were created to show how individual 
performance from each participant relates to their team 
member performance. Differences in variability exist as 
reflected in the box-plot of participant six and 17 in 
Figure 12. Generally, when comparing performances among 
teams, teams composed of members who were less consistent 
in checking messages (information management) performed 
worse than teams composed of members who were more 
consistent. Information management behaviors in Team 3 were 
more consistent and outperformed the other teams in both 
measures of performance: information currency and number of 
cells lost. 
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Individual characteristics and traits were correlated 
with individual performances to determine if relationships 
exist. From Appendix G, only two personality traits 
pessimism and extroversion appear to be related to 
individual performance. The positive relationship between 
pessimism and individual performance coincides with the 
literature suggesting that feelings of wanting to give-up 
and surrender are related to poor performance . In the game, 
as the fire became more out of controlled and less 
manageable, these individuals would be expected to withdraw 
from the game and refuse inputs from their teammates. 
Likewise, the negative relationship between the degrees of 
extroversion (1 = Very Low to 5 = Very High) and individual 
performance also agree with the literature. Extroverts are 
said to help facilitate intra-team communication and are 
generally ready to share information (Bradley & Herbert, 
1997). Thus, their individual time lag can be expected to 
decrease (i.e., their performance improves) as a result of 
being more extroverted. 

Also, the correlation matrix in Appendix H is color- 
coded to reflect which individual characteristics and 
traits were related to one another. The cells highlighted 
in red indicate a p-value < .01 for testing the null 
hypothesis of correlation. 

The second correlation matrix (See Appendix H) 
suggests that diverse teams perform differently from 
homogenous teams on two of six individual characteristic 
and trait variables, pessimism and optimism. Time lag 
improves the more similar team members are on their measure 
of optimism (r = -.786, p-value = .036). These teams 
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exhibit quicker information exchanges resulting in more 
current information (See Figure 15). Conversely, 
performance degrades the more similar team members are in 
their reports of pessimism. This finding coincides with 
the belief that negativity breeds negativity. 

Results from phase two of the analysis address the 
third question as to whether there are performance 
differences when blocking on various team member traits in 
team design. Cognitive uncertainty (CU) and the 
personality trait agreeableness are the two individual 
characteristics and traits identified for this phase of the 
analysis. Attrition, a very small sample size to start, 
and the fact that one team chose not to use any 
communication technology within their session, prevents any 
statistical analysis beyond graphs to summarize the data. 
Figure 22 shows that teams consisting of highly agreeable 
members have lower information management times than teams 
of low agreeable members. This finding is consistent with 
what the literature informs us about agreeableness and 
information exchange. Teams characterized by agreeable 
members are expected to exhibit interpersonal skills such 
as the ability to resolve conflict and communicate openly 
in a manner which promotes information exchange and 
sensemaking (Aronoff & Wilson, 1985; Neuman & Wright, 

1999). Thus, the time lag between a message being sent and 
read is expected to be lower for a team composed of non 
agreeable members. Average team time lag for the High 
Agreeable team was 26.4 seconds with a standard deviation 
of 25.9 where as the Low Agreeable team had an average time 
lag of 67.9 seconds with a standard deviation of 56.7 
seconds. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

Advocates of NCW claim that it will improve 
information sharing by "...robustly networking a force", 
thereby improving mission effectiveness. However, this 
claim has not been tested or validated. The literature is 
inconclusive in identifying individual characteristics and 
traits that would affect macrocognitive processes such as 
information sharing and sensemaking. 

To guide the analysis about NCW, the model of DMSC was 
applied and adapted. This model provided a conceptual 
framework for describing macrocognitive processes such as 
information exchange and sensemaking. Feedback loops and 
lenses presented in the model illustrate how information 
can be omitted or included in individual or team cognition. 
The lenses function as filters which help characterize 
differences in individual characteristics and traits. The 
feedback loops in the DMSC are representative of time lag 
among individuals on the teams and time lag affects on 
individual and overall performance. 

Human factors engineering (HFE) is a discipline that 
focuses on designing systems around users (i.e., user- 
centered design) and employing technology that acknowledges 
and complements human characteristics and traits. Although 
the pilot study, by its very nature, did not produce 
significant results for assessing performance differences, 
it is evident that performance differences can be explained 
by individual characteristics and traits. There are 
tremendous variations in the behaviors, expectations, and 
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mental capabilities of people. As the DMSC model depicts, 
individual characteristics and traits such as personality, 
cultural adaptability, optimism-pessimism, and responses to 
uncertainty occlude interactions within teams and 
technology. These differences, if unaccounted for in team 
design, training, or systems design, are barriers that 
obstruct both individual and team performance. 

Failure in understanding individual characteristic and 
trait differences within the DoD population creates 
disconnects between networking technologies and the 
capabilities of the men and women expected to operate 
within a network centric system. The personnel and 
training domains focus on identifying these disconnects by 
focusing on the target audience. The target audience 
includes the types of systems used (i.e., communications 
technologies and group decision support systems) and key 
statistics on the personnel pool. Assessing users on the 
basis of personality, cultural adaptability, and optimism- 
pessimism is a first step in achieving perceived advantages 
of distributed teamwork. Testing a variety of team 
compositions of military personnel will provide insight as 
to the types of people that are better suited for 
distributed teamwork. The pilot study suggests that 
extroverts and optimists are more likely to engage in 
information exchanges via networks. 

B. RECOMMENDATIONS 

Empirical evidence must be provided to facilitate the 
acceptance of NCW as an emerging theory of warfare. Basic 
and applied research must be carefully conducted to begin 
identifying and isolating individual characteristic and 
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trait variables which may jeopardize mission effectiveness 
in the Information Age. Technology solutions developed 
independent of understanding human behaviors, such as 
macrocognition in teams, invite failure. 

C. FOLLOW-ON RESEARCH 

People and their interaction with technology are the 
primary and secondary focuses of NCW. To totally integrate 
systems, additional research must apply process models of 
human technology interaction such as the DMSC to 
experimental, field, or observational settings designed to 
test facets of NCW. These kinds of dynamic models provide 
extreme possibilities in advancing our understanding of 
which human factors have the greatest affect on 
performance. 

The CSFire simulation uses asynchronous technology 
(i.e., e-mail) for coordination and communication within 
teams. Little is known about how the types of 
technologies, asynchronous or synchronous, influence the 
quality of macrocognitive processing at the individual or 
team level. Our understanding about e-mail, text-chat, 
video teleconferencing, or face-to-face communication 
systems will affect the quality of team interactions (i.e., 
communication, planning, and coordination) and must guide 
the development and deployment of NCW technologies. Future 
research should include an examination of individual 
characteristics and traits in order to further understand 
which types of personnel are more comfortable with 
employing asynchronous technologies in macrocognitive 
processes. The literature suggests that people who have 
high needs to achieve certainty may gather and hoard 
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information before contributing or making a decision. This 
behavior would undermine NCW claims of networking a 
distributed force to improve the quality and amount of 
information sharing to achieve mission effectiveness. 
Moreover, HFE research should focus on varying 
communication modalities and technologies to determine 
which communication modalities enhance information exchange 
and sensemaking, making them better at facilitating 
distributed teamwork. 

Research about human performance has traditionally 
used civilian undergraduate students. However, military 
transformation implies understanding the current population 
of users, and redeveloping tactics, techniques, and 
procedures that better enable them to achieve mission 
effectiveness. Critical information in determining 
military personnel requirements needs to be gathered. 
Therefore, human performance researchers must redefine 
their target populations and exploit service men and women 
at various military schools . 

The first tenet of NCW postulates that if a force is 
robustly networked, then they would share information and 
that this information exchange would lead to improve 
mission effectiveness. There are other human factors such 
as organizational structure, cross planning and execution 
between military versus non-military organizations, trust, 
and team cohesion, which affect team performance. 

Objective and qualitative measures of individual and team 
performance in a network environment need to be 
established. The results of individual and team measures 
could then be used to maximize human-system performance. 
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APPENDIX A. GLOSSARY AND ACRONYMS 


AACS Ability to Achieve Cognitive Structure, 

the extent to which an individual is 
able to avoid information that either 
cannot be categorized or clashes with 
their existing knowledge and/or ability 
to organize their knowledge to fit an 
already existing cognitive structure 
(Bar-Tal, 1994, p. 46) 

AWACS-AEDGE Airborne Warning and Control System - 
Agent Enabled Decision Guide 
Environment 

CA Conflict Avoidance 

C2 Command and Control 

DMSC Dynamic Model of Situated Cognition 

HSI Human Systems Integration 

HSIL Human Systems Integration Laboratory 

NCS Need for Cognitive Structure, the 

desire for clear and firm knowledge 
concerning a given topic as opposed to 
ambiguity doubt, or confusion 
(Kruglanski, 1989, as cited in Bar-Tal, 
1994, p. 46) 

NCW Network Centric Warfare, the conduct of 

military operations using networked 
information systems to generate a 
flexible and agile military force that 
acts under a common commander's intent, 
independent of the geographic or 
organizational disposition of the 
individual elements, and in which the 
focus of the warfighter is broadened 
away from individual, unit or platform 
concerns to give primacy to the mission 
and responsibilities of the team, task 
group or coalition. (Eewell & Hazen, 
2003, p. 39) 

NEO-EEI NEC - Eive-Eactor Inventory 
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OP I 


Optimism 


Pessimism Instrument 


Optimism 

Pessimism 

SABRE 

Sensemaking 

URS 

Virtual Team 


A disposition inclining one to positive 
expectations 

A disposition inclining one to negative 
expectations 

Situation Authorable Behavior Research 
Environment 

A behavior that is both internal (i.e., 
cognitive) and external (i.e. 
procedural), which allows the 
individual to construct and design 
their movement through time-space 
(Dervin, 1983, p.3) 

Uncertainty Response Scale 

a collection of individuals who 
are interdependent in their tasks, who 
share responsibility for outcomes, who 
see themselves and who are seen by 
others as an intact social entity 
embedded in one or more larger social 
systems, and who manage their 
relationship across organizational 
boundaries. (Powell, Piccoli, and Ives 
,2004, p. 241) 
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APPENDIX B. NCW DEPENDENT VARIABLE 


A. INFORMATION EXCHANGE (CURRENCY, FROM EVIDENCED BASED 
RESEARCH, 2004, P.37) 


Attribute 

Metrics 

Objective Measures 

Measnres quality in reference to criteria that are independent of the situation 

Correctness 

Correspondence vrith ground trutbcorrelation coefficient (0= no convergence, l=fnll 
convergence between individuals awareness and ground truth) 

Consistency 

Degree of ‘devlatioif from awareness gained from previous time period 

Currency 

Time lag of awareness 

Precision 

Level of grannlarity of awareness 



Fitness for Use 

Measures 

Measures quality in reference to criteria that are determined by the situation 

Completeness 

Percentage of ground truth picture included in awareness 

Accuracy 

Degree to which precision matches what is needed (0=no match, 10=high degree of matching 
between precision level needed and available) 

Relevance 

Proportion of awareness that is related to task at hand 

Timeliness 

Degree to which currency matches what is needed (0=no match, 10=high degree of matching 
between currency level needed and available) 

Uncertainty 

Confidence level (0% =uncertain, 100%= certain) or confidence interval (95%, 90%, etc.) of 
awareness 


Time 

of 

Event 

, 1 

Time 

Event 

Posted 

1 

Time 

Information 

Received 

at 

Command Ctr 

1 

Time 
Event 
Noticed 
by Staff 

L 

Time 

Event 

Significance 
Projected 
by Staff 

L 

Time 

Event 

Response 

Declared 

by 

Commander 

_ 

"^event 

Ipost 

"^receive 

- 

"^awareness "^understanding "^decision 

_ 

-- 

Time lag of awareness 





63 






























THIS PAGE INTENTIONALLY LEET BLANK 


64 



APPENDIX C. POINTS OF CONTACT 


A. NAVAL POSTGRADUATE SCHOOL 

Nita Lewis Miller, Ph.D. 

Director, Human Systems Integration Program 
Lyn R. Whitaker, Ph.D. 

Associate Professor of Operations Research 
Lawrence G. Shattuck, Ph.D. 

Co-Director, Human Systems Integration Program 
Senior Lecturer 

B. C3FIRE 

Rego Granlund, Ph.D. 

Developer of CSFire 

Professor, Linkoping Institute of Technology 
Kip Smith, Ph.D. 

Professor, Linkoping Institute of Technology 
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APPENDIX D. SURVEY RESULTS 


Personality results measured using the NEO-FFI 


Extroverts 



Openness 



Very Low Average High Very 
Low Hgh 


Agreeableness 



Neurotics 



Conscientiousness 


15 



Low High 

Bin 
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APPENDIX E. SURVEY RESULTS 


Optimism and Pessimism results using the OPI. 


Responses to the Optimism Pessimism instrument (OPi) 



Responses to Uncertainty using the URS 



Conflict Avoidance Responses 


Conflict Avoidance (CA) Scores 



Participant 
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APPENDIX F. SURVEY RESULTS 


Ability to Achieve Cognitive Structure Responses 


AACS Total 



Participants 


AACS Total 


Need for Cognitive Structure Responses 


NCS Total 



Particiants 


— NCS total/person 
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APPENDIX G. CORRELATION MATRIX OE INDIVIDUAL 
CHARACTERISTICS AND TRAITS WITH TEAM PEREORMANCE 
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APPENDIX H. CORRELATION MATRIX OE TEAM 
CHARACTERISTICS (DEGREE OE SIMILARITY) WITH 

PEREORMANCE 



Team Averaae 








Emotion 

Desire fo 

Coanitive 

Conflict 

Number 


Time Lag (in 

Optimisim 

Pessimisrr 

Extroversio 

Openness 

Agreeable 

Neuroticism 

Conscientiousness 

Uncertainty 

Chanoef 

Uncertainty 

Avoidance 

of Lost 


sec) 

simiiiarity) 

(similarity) 

n(similarity) 

similarity) 

similarity) 

(similarity) 

(similarity) 

similarity) 

similarity) 
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